IMAGE FILTER METHOD 



CROSS-REFERENCE TO RELATED APPLICATIONS 

The following patent applications disclose related subject matter: Appl. 

Nos. 09/ , filed .... ( — ). These referenced applications have a common 

assignee with the present application. 

BACKGROUND OF THE INVENTION 

The invention relates to image processing, and more particularly to image 
filtering methods and related devices such as digital and video cameras. 

There has been considerable growth in the sale and use of digital 
cameras, both still and video, in recent years. Figure 4 is a block diagram of a 
typical digital still camera which includes various image processing components, 
collectively referred to as an image pipeline. Color filter array (CFA) 
interpolation, gamma correction, white balancing, color space conversion, and 
JPEG (or MPEG for video) compression-decompression constitute some of the 
key image pipeline processes. 

In DCT-based video/image compression, such as MPEG or JPEG, a low 
bit rate (high compression) for efficient transmission or storage is known to cause 
annoying artifacts, such as mosquito-noise, block noise, etc. In order to reduce 
these artifacts, preprocessing of input images is required. However, conventional 
linear filtering often reduces the detail clarity as well as the artifacts in the output 
signal. However, the size of such filters becomes large when the desired 
characteristics are demanding, and this results in prohibitively large circuit size. 

Infinite impulse resonse (IIR) filtering is often used in acoustical signal 
processing. However, it is little used in image processing due to its side effects, 
which are often imperceptible in sound but apparent in images. 

Filtering using the matching method compares input signals with a stored 
database and outputs appropriate signals. Although this method works well in 
some situations, the output quality can be low if the database does not match the 
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input. Also, this method consumes large amounts of memory and computational 
power. 

SUMMARY OF THE INVENTION 

The present invention provides image preprocessing methods and 
systems with filtering using estimates of the power spectrum distribution of the 
input image by the auto-correlation and applies appropriate filtering accordingly. 

This has advantages including enhanced quality of DCT-based image 
compression. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The drawings are heuristic for clarity. 

Figures 1a-1b are a flow diagram for a preferred embodiment method and 
a preferred embodiment digital camera system. 

Figures 2a-2b illustrate artifacts. 

Figures 3-5 show compression for various textures. 

Figures 6-7 show preferred embodiment metric characteristics. 

Figures 8a-8c illustrate preferred embodiment metric distortion indication. 

Figure 9 shows preferred method pre-processing and compression. 

Figures 10-13 compare preferred embodiment methods with prior art 
methods. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 



1 . Overview 

Preferred embodiment image filtering methods include two steps: the first 
step evaluates the local characteristics of the image, and the second applies 
filtering to the local area according to the result of evaluation. In particular, 
boundaries between bright and dark planes show the most annoying artifacts 
when DCT processed with high-frequency quantization; so the preferred 
embodiments locally smooth such boundaries while leaving areas with low- 
variability intensity and areas with high-variability intensity unsmoothed. The 
preferred embodiments detect boundaries between bright and dark by noting that 
the power spectrum of such boundaries (in the continuous variable case) decays 
roughly like 1/w where u) is the spatial frequency, whereas the low-variability 
power spectrum decays roughly like 1/w 2 or faster, and the high-variability power 
spectrum is roughly constant. 

Figure 1a is a flow diagram for a preferred embodiment method. The 
methods allow a relatively simple computation, a modified correlation coefficient, 
to determine pre-processing filtering to suppress DCT-base compression 
artifacts. 

Preferred embodiment digital image systems (such as cameras) include 
preferred embodiment image pre-processing filtering methods. Figure 1b shows 
in functional block form a system (digital still camera) which incorporates 
preferred embodiment methods as shown in the JPEG compression block. The 
functions of preferred embodiment systems can be performed with digital signal 
processors (DSPs) or general purpose programmable processors or application 
specific circuitry or systems on a chip such as both a DSP and RISC processor 
on the same chip with the RISC processor as controller. Further specialized 
accelerators, such as CFA color interpolation and JPEG encoding, could be 
added to a chip with a DSP and a RISC processor. Captured images could be 
stored in memory either prior to or after image pipeline processing. The image 
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pipeline functions could be a stored program in an onboard or external ROM, 
flash EEPROM, or ferroelectric RAM for any programmable processors. 

2. DCT-based compression artifacts 

This section briefly reviews artifacts in DCT-based compression, and the 
analysis of the origin of artifacts is described. 

Figures 2a-2b show examples of artifacts in 8x8 block DCT-based 
compression. Figure 2a is the original (uncompressed) image and Figure 2b is 
the compressed (JPEG) image. Various distortions are visible in Figure 2b. By 
closely examining distortion in various images, including this figure, the following 
characteristics have been discovered. 

(a) Artifacts are very small where spatial variation is small (see box "a" in 
Figures 2a-2b). 

(b) Distortion is large at the boundary of bright plane and dark plane (see box 
"b" in Figures 2a-2b). 

(c) Artifacts exist, but are not noticeable, in complex texture, (see box "c" in 
Figures 2a-2b). 

A schematic picture of each (intensity) signal pattern (horizontal or vertical 
through one of the boxes) is shown in Figure 3, and their corresponding DCT 
coefficient signals are shown in figure 4. Here, the signal is x(n) (n = 0, 1, 7), 
and the DCT coefficients are defined as. 



Also, the corresponding compressed sptial signals (inverse DCT after 
quantization) are shown in Figure 5. 

By comparing Figures 3 and 5, it is obvious that the pattern (b) generates 
the most annoying distortions. Observation of DCT coefficients of each signal 
pattern (Figure 4) indicates that this symptom is caused by the size of the 
quantization level. In DCT-based compression, the quantization level is larger in 
high frequency regions. Therefore, high frequency components are most 
affected. From Figure 4, the following tendencies can be found for the patterns. 




(1) 



TI-35909 



Page 4 



- Pattern (a): High frequency coefficients are negligibly small. 

- Pattern (b): Coefficients gradually degrades as frequency increases. 

- Pattern (c): High frequency coefficients are large. 

In pattern (b) the high frequency components are distorted by the quantization 
level because the coefficients are small. However, pattern (a) shows small 
distortion because the coefficients are negligible anyway. On the other hand, 
coefficients in pattern (c) are larger than the quantization level, resulting in small 
distortion. Apparently, this is the reason why the distortion is most obvious in 
pattern (b). Note that if the DC component were removed, then pattern (a) would 
be very small at all frequencies, pattern (b) would be primarily low frequencies, 
and pattern (c) would be primarily high frequencies. 

Based on the above, the preferred embodiment method strategy is. 

(1) Find the pixels with surrounding blocks having patterns similar to the pattern 
(b) in Figures 3-4. 

(2) Apply low pass filtering at these pixels. 

In the following sections, each step is explained in detail. 

3. Power spectrum and auto-correlation 

In this section, the mathematical analysis of power spectrum is explained 
for continuous variables. Then, a metric to measure the shape of the power 
spectrum, which underlies the preferred embodiment methods, is introduced. 

The schematic picture of a power spectrum is shown in figure 6. It is 
assumed that the spectrum is confined to -oj, a < oj < ujth- Also, the spatial 
average of the signal is assumed to be zero; that is, remove any DC component, 
prior to the following calculations. 

In order to evaluate the distribution of the spectrum, introduce a metric, J, 
which measures the distribution of a power spectrum: 




(2) 
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where f(co) is an arbitrary function which shows positive values near oo=0, and 
negative values near co=co th (see Figure 6). If the power spectrum distribution lies 
primarily in the low frequency region, the combined signal S(co)f(co) distribution 
lies in the low frequency regions with positive values, and J will be positive (see 
Figure 7). Contrarily, if the power spectrum distribution lies primarily in the high 
frequency regions, the combined signal S(co)f(co) distribution lies in the high 
frequency regions with negative values, and J will be negative. In short, 

-- primarily low frequency S(co) implies positive J. 

- primarily high frequency S(co) implies negative J. 
Thus the immediate objective is to evaluate J to find the distribution of the power 
spectrum. 

With f(o})=co 0 2 -o) 2 (illustrated in Figure 6), / becomes: 



Note that the auto-correlation function is the Fourier transform of the power 
spectrum; that is: 




(3) 



Next, introduce the auto-correlation function, R^, as follows. 




(4) 




(5) 



Also, the second derivative of the auto-correlation function is 




(6) 



Thus the second term in / can be written as. 




Also, the denominator in equation (2) can be written as. 




(8) 



Thus / becomes 



I = 2xo) 0 2 R xx (Q)+2xR" xx {6) 



(9) 
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Combining equation (8) and equation (9) yields: 

J = — L- = 2xa> 0 2 +^\ (10) 

Hence, the evaluation of the power spectrum distribution reduces to the 
evaluation of the auto-correlation function. 

The above equations were carried out for continuous time signals. Thus 
adapt equation (10) for discrete time signals. With discrete time signals, the 
auto-correlation function is written as 

/U0=2>M«+0 

n 

In this case, approximate the derivatives by differences: 

C(r)«fc(r + l)-^(r)}-{^(r)-i?„(r-l)} (12) 

Thus, 

*L(o) - {Rjo)~ R a (- i)}= 2(^(1)- rM (13) 

Then J is approximated by 

'*,(!) ^ 



J ^27to) 0 2 + 2 



2{p-A) (14) 



where 



p = jM y A = i-^ (15) 



Here, A is a parameter set by the crossover frequency wo, and p is the auto- 
correlation coefficient. Thus evaluation of spectrum distribution metric reduces to 
computation of the auto-correlation coefficient p. If p is small (J negative), then 
the spectrum distribution is primarily in the high frequency region. If p is large (J 
positive), the spectrum distribution lies primarily in the low frequency region. 

From Figure 4, it can be seen that pattern (b) shows large low frequency 
components and small high frequency components. Therefore, positive p means 
that the signal is close to pattern (b), which shows the largest distortion in DCT- 
based compression. On the other hand, pattern (c) will show negative/?. 
However, pattern (a) may be mistaken as pattern (b), because both may have 
positive p. In order to distinguish pattern (a) from (b), p is modified as follows: 
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Here, S is an arbitrary number smaller than average Rxx(O). If ***(o)» s , p is the 
same as the original. If the signal is close to pattern (a) in Figure 4, its auto- 
correlation function will be close to zero after removal of the DC component, so p 
will also be near zero due to the S term. Using this modified value distinguishes 
pattern (b) from pattern (a) without changing other characteristics. Therefore, the 
preferred embodiment methods use modified p in the following. 

Figures 8a-8c provide an example of/?. Figure 8a is the original image, 
and Figure 8b is the corresponding p distribution computed using intervals of nine 
horizontal pixels about a pixel but with the DC component removed prior to 
computation. In Figure 8b, negative p is represented by black, and positive p 
white. Note that p is positive at the boundary between white plane and dark 
plane (box 1 ), while p is negative at the stripe (box 2). Figure 8c shows the 
JPEG compressed image resulting from the image of Figure 8a. The positive 
(white) p in Figure 8b corresponds to artifacts in Figure 8c, indicating that p 
expresses the likelihood of distortion. 

The difference between the preferred embodiment method and the 
conventional edge detection technique should be emphasized. In the 
conventional technique, the stripe pattern is considered as a group of edges, just 
like the boundary between two planes. On the other hand, the preferred 
embodiment method distinguishes the boundary from the stripe pattern. 

In summary, metric p represents the distribution of the power spectrum 
and represents the likelihood of distortion in DCT-based compression. 

4. First preferred embodiment 

Figure 1 is a flow diagram of first preferred embodiment image filtering 
methods which include the following steps. 
(1) Compute a modified auto-correlation coefficient, p = ^(i)/(^(o)+£), in the 

local area near the pixel of interest in the horizontal direction. The area for 
calculation is determined by the computational level allowed; usually, an interval 
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of 7 to 9 pixels is enough. First, subtract the DC component (the average), and 
then compute R a (\) and fljo) . 

(2) Determine the intensity of the filtering according to p, so that filtering is 
applied to places with positive p. For example, set the filtering intensity 
proportional to {p- p, h ), where p lh is a user defined parameter; then apply low 
pass filtering according to the intensity. More explicitly, start with the simple low 
pass filter x(n) -^y(n) = [x(n-\) + 2x(n) +x(n+l)]/4 and then define the overall 
filtering to be x(n) — ► (1-0 x(n) + (i) y(n) where the intensity i = 5 {p-p th )IA. 

(3) Repeat steps 1 and 2 for each pixel in the image. 

(4) Perform steps 1 through 3 for the vertical direction. 

If the image is in color, the filtering is applied to each color. Further; if the image 
is in Y-U-V or Y-Cr-Cb format, then an alternative would be to only filter the 
luminance Y. 

5. Experimental 

Figure 9 is a JPEG compressed image of the test pattern in Figure 8a after 
filtering with the preferred embodiment method. The pixel values were in the 
range 0-255, a 9-pixel interval was used for the correlations, 8 = 64 (if the pixel 
values were normalized to 0<xfw;<1 , then S ~ 0.001 ), and p lh = 0.0. The 
compression ratio is very close to that of Figure 8c (Figure 8c: 14.9%, Figure 9: 
14.7%). At boundaries such as the area in box 1, the distortion is suppressed 
compared to Figure 8c. However, the stripe in box 2 is still clear. 

Figure 10 shows an example of a natural image. Figures 11-13 show 
JPEG compressed images of original image Figure 10, without pre-processing, 
with pre-processing using the preferred embodiment method, and with pre- 
processing using the conventional method, respectively. All three JPEG images 
have very similar compression ratios (19.3%-19.7%). 

Figures 12-13 show that, both preferred embodiment and conventional 
filtering methods reduce artifacts caused by DCT-based compression. However, 
details are much clearer in the preferred embodiment method. For example, the 
artifacts near a tree are reduced in both pictures (see Figures 12a, 13a). On the 
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other hand, loss in detail clarity is minimal in figure 12b (preferred embodiment 
method), compared to Figure 13b (conventional method). The pattern of leaves 
is visible in Figure 12b; however, it is blurred and not visible in Figure 13b. 
Figures 12c and 13c also show the same tendency. The artifacts around the 
plate are reduced in both Figures 12c and 13c. However, the letters in Figure 
13c are hardly readable because of blurring, while the letters in Figure 12c are 
still clear. Figure 12d and Figure 13d show another example. In Figure 12d the 
white line in the center is solid; however, in Figure 13d it is barely observable. 

The above results show the superiority of the preferred embodiment 
method over the conventional linear filtering as a pre-processing technique in 
DCT-based compression. In short, the preferred embodiment method has 
following merits. 

(a) Reduction of artifacts in DCT-based compression 

(b) Preservation of image detail 

6. Modifications 

The preferred embodiments may be modified in various ways while 
retaining one or more of the features of pre-processing filtering derived from 
modified auto-correlations. 

For example, the 7-9 pixel interval size for the auto-correlation could be 
varied to other sizes. The parameters such as d,p t h, and i could be varied. 
Differing functions/fwj lead to replacing p with other combinations of derivatives 
of the auto-correlation; and so forth. 
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